Skip to content

PERF: ExpandingGroupby #37064

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 23 commits into from
Oct 12, 2020
Merged

PERF: ExpandingGroupby #37064

merged 23 commits into from
Oct 12, 2020

Conversation

mroeschke
Copy link
Member

  • tests added / passed
  • passes black pandas
  • passes git diff upstream/master -u -- "*.py" | flake8 --diff
  • whatsnew entry
       before           after         ratio
     [89cc5bf9]       [6af880e7]
     <master>         <feature/groupby_expanding_indexer>
-      45.5±0.6ms      3.78±0.04ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('Series', 'int', 'max')
-      46.5±0.5ms      3.86±0.06ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('Series', 'int', 'mean')
-      45.4±0.4ms      3.74±0.05ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'int', 'min')
-      45.5±0.4ms      3.73±0.06ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('Series', 'int', 'min')
-      46.6±0.3ms      3.80±0.01ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('Series', 'int', 'median')
-        46.7±1ms      3.77±0.05ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'int', 'max')
-      46.3±0.6ms      3.73±0.02ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'int', 'sum')
-      46.8±0.3ms       3.76±0.1ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('Series', 'int', 'sum')
-      46.7±0.5ms      3.75±0.02ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'int', 'mean')
-      47.2±0.3ms      3.78±0.04ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'int', 'median')
-      51.2±0.7ms       3.89±0.1ms     0.08  rolling.ExpandingMethods.time_expanding_groupby('Series', 'int', 'std')
-      51.3±0.3ms      3.78±0.02ms     0.07  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'int', 'std')
-      62.6±0.6ms      3.98±0.02ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('Series', 'int', 'count')
-      62.2±0.7ms      3.91±0.03ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'int', 'count')
-      64.3±0.7ms      3.98±0.05ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('Series', 'float', 'sum')
-      63.5±0.7ms      3.92±0.01ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'float', 'max')
-      63.6±0.6ms      3.91±0.03ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('Series', 'float', 'min')
-      63.2±0.7ms      3.86±0.02ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'float', 'min')
-        64.2±1ms      3.89±0.04ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('Series', 'float', 'max')
-      64.8±0.8ms      3.91±0.06ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'float', 'sum')
-      65.2±0.6ms      3.92±0.05ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'float', 'mean')
-      65.2±0.9ms      3.90±0.05ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('Series', 'float', 'median')
-      65.5±0.5ms      3.91±0.02ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'float', 'median')
-      65.3±0.2ms      3.86±0.01ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('Series', 'float', 'mean')
-      69.4±0.8ms      4.01±0.07ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('Series', 'float', 'std')
-      69.8±0.6ms      3.93±0.01ms     0.06  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'float', 'std')
-      80.1±0.2ms       4.24±0.1ms     0.05  rolling.ExpandingMethods.time_expanding_groupby('DataFrame', 'float', 'count')
-      80.5±0.9ms      4.14±0.02ms     0.05  rolling.ExpandingMethods.time_expanding_groupby('Series', 'float', 'count')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

Matt Roeschke added 20 commits September 27, 2020 12:09
Add groupby expanding indexer
c1d6af1
Impliment apply for ExpandingGroupby
37cefba
Merge remote-tracking branch 'upstream/master' into feature/groupby_e…
6918c28 
…xpanding_indexer
Get most tests to work except numerical precision test
77deb50
Make signature of GroupbyExpanding indexer match base class
43d2682
Share groupbyindexer implementation
409bda8
Merge remote-tracking branch 'upstream/master' into feature/groupby_e…
1146a2f 
…xpanding_indexer
Merge remote-tracking branch 'upstream/master' into feature/groupby_e…
a04702e 
…xpanding_indexer
Merge remote-tracking branch 'upstream/master' into feature/groupby_e…
542444d 
…xpanding_indexer
Remove need to select variable algorithm
0dacaa5
Merge remote-tracking branch 'upstream/master' into feature/groupby_e…
7683214 
…xpanding_indexer
Create new subclass for shared window groupby algorithms
ed211e7
Merge remote-tracking branch 'upstream/master' into feature/groupby_e…
6885d14 
…xpanding_indexer
Add expanding groupby benchmark
57471be
Merge remote-tracking branch 'upstream/master' into feature/groupby_e…
bff2316 
…xpanding_indexer
Merge remote-tracking branch 'upstream/master' into feature/groupby_e…
6af880e 
…xpanding_indexer
Add benchmarks
608e7d6
Remove unneeded args passed to _apply
859541e
Remove unnecessary poping of kwargs
fc43e23
Add whatsnew
bab0a1d
@mroeschke mroeschke added Performance Memory or execution speed performance Window rolling, ewma, expanding labels Oct 12, 2020
@mroeschke mroeschke added this to the 1.2 milestone Oct 12, 2020
mroeschke
mroeschke commented Oct 12, 2020
View reviewed changes
pandas/_libs/window/aggregations.pyx
@@ -302,7 +302,7 @@ cdef inline float64_t calc_var(int64_t minp, int ddof, float64_t nobs,
result = ssqdm_x / (nobs - <float64_t>ddof)
# Fix for numerical imprecision.
# Can be result < 0 once Kahan Summation is implemented
if result < 1e-15:
if result < 1e-14:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xref #37051, needed to change for an expanding groupby test

@mroeschke mroeschke mentioned this pull request Oct 12, 2020
Open
32 tasks
jreback
jreback requested changes Oct 12, 2020
View reviewed changes
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we expose the GroupIndexer in pandas.api.indexers? (if not then this is not public yet)

@mroeschke
Copy link
Member Author

do we expose the GroupIndexer in pandas.api.indexers?

No. I don't think GroupByIndexer should be public though

@jreback jreback merged commit ca27cca into pandas-dev:master Oct 12, 2020
@jreback
Copy link
Contributor

jreback commented Oct 12, 2020

great thanks

@mroeschke mroeschke deleted the feature/groupby_expanding_indexer branch October 12, 2020 17:08
JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Oct 26, 2020
@mroeschke @JulianWgs
PERF: ExpandingGroupby (pandas-dev#37064)
8b7c006
kesmit13 pushed a commit to kesmit13/pandas that referenced this pull request Nov 2, 2020
@mroeschke
PERF: ExpandingGroupby (pandas-dev#37064)
355cbb1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jreback jreback jreback approved these changes

Assignees
No one assigned
Labels
Performance Memory or execution speed performance Window rolling, ewma, expanding
Projects
None yet
Milestone
1.2
Development

Successfully merging this pull request may close these issues.
2 participants
@mroeschke @jreback